Model-Based Hierarchical Clustering

We present an approach to model-based hi­ erarchical clustering by formulating an ob­ jective function based on a Bayesian anal­ ysis. This model organizes the data into a cluster hierarchy while specifying a complex feature-set partitioning that is a key compo­ nent of our model. Features can have either a unique distribution in every cluster or a com­ mon distribution over some (or even all) of the clusters. The cluster subsets over which these features have such a common distribu­ tion correspond to the nodes (clusters) of the tree representing the hierarchy. We apply this general model to the problem of docu­ ment clustering for which we use a multino­ mial likelihood function and Dirichlet priors. Our algorithm consists of a two-stage pro­ cess wherein we first perform a flat clustering followed by a modified hierarchical agglom­ erative merging process that includes deter­ mining the features that will have common distributions over the merged clusters. The regularization induced by using the marginal likelihood automatically determines the op­ timal model structure including number of clusters, the depth of the tree and the subset of features to be modeled as having a com­ mon distribution at each node. We present experimental results on both synthetic data and a real document collection.